Computer software has become increasingly complex and sophisticated. In addition, software has increasingly found use in mission-critical applications and other applications where human lives and/or livelihoods are at stake. In these applications, the reliable and failsafe manner in which the software executes under all possible operating conditions is often a paramount concern. As a result, during development of such software, often the tasks of debugging and testing the software prior to its final release are as important as the tasks associated with generating the actual program code.
One important technique used in connection with testing computer software under development is coverage analysis. In particular, software is often tested by subjecting the software to various test conditions and executing the software to attempt to identify any errors that may arise under those test conditions. Ideally, all possible conditions that a piece of software could be subjected to during use by customers would be tested. Furthermore, ideally each and every line of code in the software would be executed at some point during testing to ensure that all possible conditions have been tested.
Coverage analysis is used to determine various metrics regarding the thoroughness of the testing procedures applied to software under development. As such, coverage analysis essentially determines the quality of the tests used to exercise software under development, as opposed to the quality of the actual software.
Coverage analysis may be used, for example, to determine whether all statements in a software program are executed during execution of the program, and if not, which statements were missed during execution of the program. Coverage analysis may also be used to detect whether all branches from a conditional or decision statement are taken during execution of a program. Other data-related analysis may also be performed, e.g., to determine whether each entry in a data array has been referenced, or whether different outcomes of relational operators or boolean operators are tested.
A basic premise behind coverage analysis is that, if a program is thoroughly exercised for all possible test conditions and all possible paths and potential exceptions are tested, and yet no errors are found, there is a low probability that errors are present in the final program.
One limitation with conventional coverage analysis and the various automated tools utilized to perform such analysis, however, is that no consideration is often taken as to the interaction between a program and various resources relied upon during execution of that program. One notable example is the interaction of a program with a backend database.
As an example, consider a program that accesses a database file. The program may access the file and process certain records, but the percentage of records from the file accessed may be small. A conventional coverage tool would typically notify a developer that the code has been covered because the program statements that access the database are executed at least once, yet the picture is incomplete since additional testing could be performed by accessing a greater percentage of records and running more data through the program. In some applications, simply testing the fact that a program accesses a database file may be sufficient for testing purposes. In other applications, however, a greater indication of the reliability of the testing protocols may result where additional data is processed by the program when interacting with a particular database file.
Another drawback of conventional coverage tools with respect to programs that access databases results from the lack of coverage analysis as to any specific attributes stored for the files in a database. Consider, for example, a program that is attempting to calculate percentages by dividing some value by the value stored in a particular field of one or more records in a database file. One potential error that could occur during execution of such a program is a divide by zero error if the value stored in the field of any processed records is zero. Unless this condition is actually tested, e.g., by attempting to process a record having a zero value in the appropriate field, or by attempting to store a zero value in the field of a record, one would not uncover the divide by zero error.
As such, for programs that access databases, conventional coverage analysis techniques and tools often present an incomplete picture, which can dilute the value of performing coverage analysis on such programs. Therefore, a significant need exists in the art for a manner of addressing coverage analysis of the database-related aspects of computer programs.